Skip to content

Feature: Add diskquota extension to gpcontrib#1490

Merged
tuhaihe merged 327 commits intoapache:mainfrom
tuhaihe:add-diskquota
Feb 27, 2026
Merged

Feature: Add diskquota extension to gpcontrib#1490
tuhaihe merged 327 commits intoapache:mainfrom
tuhaihe:add-diskquota

Conversation

@tuhaihe
Copy link
Member

@tuhaihe tuhaihe commented Dec 12, 2025

Fixes #ISSUE_Number

What does this PR do?

Type of Change

  • Bug fix (non-breaking change)
  • New feature (non-breaking change)
  • Breaking change (fix or feature with breaking changes)
  • Documentation update

Breaking Changes

Test Plan

  • Unit tests added/updated
  • Integration tests added/updated
  • Passed make installcheck
  • Passed make -C src/test installcheck-cbdb-parallel

Impact

Performance:

User-facing changes:

Dependencies:

Checklist

Additional Context

CI Skip Instructions


@tuhaihe tuhaihe force-pushed the add-diskquota branch 2 times, most recently from c916aae to 92de2e8 Compare December 17, 2025 09:39
@Smyatkin-Maxim Smyatkin-Maxim self-requested a review December 19, 2025 13:08
Copy link
Contributor

@leborchuk leborchuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked it on my development ubuntu installation, all simple tests works fine

Here the output from examples

postgres=# select * from diskquota.show_fast_schema_quota_view;
 schema_name | schema_oid | quota_in_mb | nspsize_in_bytes
-------------+------------+-------------+------------------
 s1          |      17157 |           1 |            98304

Copy link
Contributor

@my-ship-it my-ship-it left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contributions that enable Cloudberry to support more components. Is it possible for us to retain the previous commit history?

@tuhaihe
Copy link
Member Author

tuhaihe commented Jan 26, 2026

Thank you for your contributions that enable Cloudberry to support more components. Is it possible for us to retain the previous commit history?

NP. Let me bring them back for a complete commit history.

@tuhaihe tuhaihe force-pushed the add-diskquota branch 3 times, most recently from 8270730 to de212a4 Compare January 26, 2026 06:28
@tuhaihe
Copy link
Member Author

tuhaihe commented Jan 26, 2026

Thank you for your contributions that enable Cloudberry to support more components. Is it possible for us to retain the previous commit history?

Hi @my-ship-it PTAL again. Thanks!

@tuhaihe tuhaihe force-pushed the add-diskquota branch 2 times, most recently from 812f1cf to fec5729 Compare February 27, 2026 02:24
higuoxing and others added 17 commits February 27, 2026 17:30
We have to create database 'diskquota' manually before running
regression tests. This change helps resolve it by creating the
database in the init stage.
* Fix stack full issue when flush to table_size

* Restore optimizer after flushing

* Resolve deadlock and add expected test result
Historically, diskquota fetches tables' size by applying pg_table_size()
on a given oid list. It may invoke pg_table_size() several times in one
transaction. In order to tolerate exceptions, we wrap it with PG_TRY()
and consume any errors throwed by pg_table_size() using FlushErrorState().
It has the following 2 potential risks.

1. Deadlock: Considering we have a table 't1' (reloid=oid1) and it has a
   partition 't1_1_prt_11' (reloid=oid2).
   When diskquota_fetch_table_stat(1, '{oid2, oid1}'::oid[]) is called
   in session 'A', this UDF will invoke pg_table_size() on 't1_1_prt_11'
   and 't1' respectively. When pg_table_size('t1_1_prt_11') throws an
   exception, the AccessShareLock acquired from session 'A' will be held
   until the whole transaction ends. At the same time, when another
   session 'B' is performing 'ALTER TABLE t1', it will acquire the
   AccessExclusiveLock on 't1' and 't1_1_prt_11' respectively. As a result,
   the session 'A' is holding the AccessShareLock on 't1_1_prt_11' and is
   waiting for AccessShareLock on 't1'; the session 'B' is holding the
   AccessExclusiveLock on 't1' and is waiting for the AccessExclusiveLock
   on 't1_1_prt_11'.

2. Error message is missing. We should preserve the error message by calling
   CopyErrorData() before FlushErrorState() and elog(WARNING, error_messgae).

Co-Authored-by: Xuebin Su <sxuebin@vmware.com>
Co-Authored-by: Hao Zhang <hzhang2@vmware.com>
Currently, tests for appendonly relations is missing. This patch adds
one test for that.

Co-Authored-by: Xuebin Su <sxuebin@vmware.com>
Co-Authored-by: Hao Zhang <hzhang2@vmware.com>
* Support get_relation_size_by_relfilenode()
add unit_test for calculating relation size by relfilenode

* Add UDFs to allow checking size of a table

Co-authored-by: hzhang2 <hzhang2@vmware.com>
Co-authored-by: Xing GUO <higuoxing@gmail.com>
Co-authored-by: hzhang2 <hzhang2@vmware.com>
This patch introduces isolation2 test framework to diskquota. We are
able to write concurrent test cases for diskquota in future.
* Fix incorrect relation size for AO tables

* Rename function to make it more consistent

* Remove try-catch when stat a file

* Add concurrent write test case for AO table
1. add relation_cache_lock
2. add relation_cache and relid_cache in SHM
3. add relation_cache test case

Co-authored-by: hzhang2 <hzhang2@vmware.com>
Co-authored-by: Xuebin Su (苏学斌) <sxuebin@vmware.com>
Co-authored-by: Xing Guo <higuoxing+github@gmail.com>
…_committed_relation_from_cache() (#96)

Co-authored-by: hzhang2 <hzhang2@vmware.com>
Co-authored-by: Xuebin Su (苏学斌) <sxuebin@vmware.com>
Co-authored-by: hzhang2 <hzhang2@vmware.com>
* Fix test case bug:
Drop test table after testing diskquota.relation_size(), because the relation_entry of the table in this test case will affect the next test case.

Co-authored-by: hzhang2 <hzhang2@vmware.com>
* Add support for dispatching blockmap to segment servers.

This patch adds support for dispatching blockmap to segment servers. The
basic idea is that, we iterate over the active relations' oid, check
that whether the relation's owner/tablespace/namespace is in one of the
blockmap entries dispatched from diskquota worker from QD. If the
relation should be blocked, we then add its relfilenode together with
the relfilenodes of toast relation, toast index relation, appendonly
auxiliary relations, appendonly auxiliary relations' indexes to the
global blockmap on segment servers. This patch is a prerequisite of the
hard-limit feature.

This patch also introduces diskquota.blockmap view. User is able to
query the current blocking status by 'SELECT * FROM diskquota.blockmap'.

Co-authored-by: Xuebin Su <sxuebin@vmware.com>
Co-authored-by: Hao Zhang <hzhang2@vmware.com>
This PR fixes incorrect JOIN condition in test_blackmap.sql. The
relfilenodes are not always different across segments. Hence, we should
add an additional JOIN condition to the test case or it will produce
unstable results.
Add calculate_table_size() to calculate any table's size, including uncommitted table.
1. Monitor active table in master.
2. Replace relation_open() with SearchSysCache(), pg_appendonly and pg_index to fetch Form_pg_class of table and index, to avoid DEADLOCK. 
3. Use DiskQuotaRelationCacheEntry to calculate table size, instead of pg_table_size(), to avoid DEADLOCK.

Co-authored-by: hzhang2 <hzhang2@vmware.com>
Co-authored-by: Xuebin Su (苏学斌) <sxuebin@vmware.com>
Co-authored-by: Xing Guo <higuoxing+github@gmail.com>
This patch adds support for adding uncommitted relations to blackmap on
segment servers. Most of the codes share similar logic with adding
committed relations to blackmap. Test will be added in the next
following commits.
zhrt123 and others added 28 commits February 27, 2026 17:31
Change the CI pipeline to release diskquota for GP7.

TODO:
- enable regress test on CI.
- enable activatestandby test for GP7 on CI.
- fix regress test for GP7. The view in GP7 will be treated as a relation by diskquota. After `CREATE EXTENSION`, the test case should execute `SELECT diskquota.init_table_size_table()` to make the diskquota.state ready on GP7.
Fix the following issues:

diskquota cannot correctly print a warning when calling
`create extension` in a non-empty database. The diskquota.state shows
whether the current database is empty. Previously, we updated the
status to diskquota.state during `create extension` and queried the
status in UDF `diskquota.diskquota_start_worker`. When querying the
database status, the relations whose `relkind in ('v', 'c', 'f')` are
skipped, while all relations are filtered when updating the database
status. This patch merges the two SQL statements to `INSERT RETURNING`
to solve this problem.
Benefit: we won't need to upgrade the minor version of diskquota when
changing the statement.

Remove useless test `test_update_db_cache`. The behavior of the
bgworker scheduler is changed, so this test is no longer helpful.

Update the upgrade test script `2.2_set_quota`. Since the above issue,
there should be `select diskquota.init_table_size_table()` in
`2.2_set_quota`.

Add upgrade test for gp7.

Fix upgrade test for gp6.
Fix the standby test on GP7.
All tables in diskquota use primary keys. Unique index tables are created for primary keys.
Unique index tables can not be created for tables with the option appendonly=true. This 
issue only occurs in gp6 since `set gp_default_storage_options='appendonly=true';` is 
disabled in gp7. Solution: add `appendonly=false` to each table with the primary key.

Co-authored-by: Evgeniy Ratkov <e.ratkov@arenadata.io>
During the upgrade procedure, when diskquota.so is not in 
shared_preload_libraries, check whether the current version 
is the final version:
- If yes, throw an error to the user.
- Otherwise, return from _PG_init().

Co-authored-by: Sasasu <i@sasa.su>
This GUC shows the max number of database diskquota can monitor.
- Fix the compilation issue caused by GP7. `libpq.so` needs to be linked. 
  See https://github.com/greenplum-db/gpdb/pull/16234.
- Add GP7 jobs back to the pipeline.
`diskquota.max_quotas` shows the max number of quotas on the cluster.
In the current version of diskquota, quotas means:
- role
- schema
- role_tablespace
- schema_tablespace

The number of quota =
       role + schema + (role * tablespace) + (schema * tablespace).
)

Fix the bug caused by #220: After the user removes any extension, the bgworker 
in the current database will be stopped.
Co-authored-by: Chen Mulong <chenmulong@gmail.com>
Co-authored-by: Xing Guo <admin@higuoxing.com>
Merge all types of quota info maps into one in each bgworker.
The package name is incorrect when releasing package in Rocky9/RHEL9
platforms (<package>-<version>-unknown_x86_64.tar.gz). This patch fix
it to <package>-<version>-rhel9_x86_64.tar.gz.

Cherry-pick https://github.com/pivotal/timestamp9/pull/41
- DDL message is only used on master, so it is unnecessary to allocate the
memory on segments.
- The launcher shmem should not be initialized on segments.
refresh_rejectmap looks for a tuple using SearchSysCacheCopy1 which retrieves
a copy of the tuple allocating memory for it. However, refresh_rejectmap didn't
free these tuple copies after use. If lots of oids were passed, diskquota could
work incorrectly because of huge memory leak. This patch frees these tuples and
prevents memory leaks.
fix: update actions

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Diskquota calculates sizes and stores information in the diskquota.table_size
table periodically with a pause in diskquota.naptime, 2 seconds by default.
If we restart the cluster during this pause, then diskquota will lose all
changes that have occurred since the last save to the diskquota.table_size
table. We could create temporary tables, wait when it will be flushed to
diskquota.table_size table, restart the cluster, and diskquota would remember
the information about the temporary tables. Or we could delete the tables,
restart the cluster, and again diskquota will remember information about the
deleted tables. This happens because at the start of the cluster, diskquota
remembers all the information written to the diskquota.table_size table,
but does not check that some tables may have already been deleted.

As a solution, we invalidate diskquota.table_size during diskquota
worker start in addition to pg_class validation.

The remaining problem: the incorrect table size cannot be refreshed until 
the corresponding table becomes an active table.

Solution: call diskquota.init_table_size_table().
Cleanup the invalid files after importing diskquota extension from upstream.

The specific changes include:
1. Remove .github directory: The upstream GitHub Actions workflows are
   no longer applicable in the Cloudberry repository.
2. Remove .gitmessage, .editorconfig, .clang-format: Code style and
   commit templates should follow Cloudberry's main repository standards.
3. Remove SECURITY.md: Legacy security policy.
4. Remove concourse directory: Legacy CI scripts.

This cleanup makes the extension structure cleaner and more consistent
with other contrib modules.
Integrate diskquota extension into Apache Cloudberry build system and
adapt the codebase for Cloudberry 2.0+ (PostgreSQL 14 based).

Main changes:

Build system integration:
* Add new Makefile for building with Cloudberry source tree
    ```
    make
    make install
    make installcheck
    make clean
    ```
* Update gpcontrib/Makefile to include diskquota in build and installcheck
* Simplify CMakeLists.txt by removing GP6 version conditionals
* Add PG_SRC_DIR availability check for isolation2 tests

Code modernization (remove GP6 compatibility code):
* Remove GP_VERSION_NUM < 70000 conditionals throughout codebase
* Replace deprecated APIs:
    heap_open -> table_open,
    heap_beginscan_catalog -> table_beginscan_catalog,
    heap_endscan -> table_endscan, etc.
* Replace init_ps_display() with set_ps_display() for process status
* Replace StrNCpy() with snprintf() for safer string handling
* Remove WaitForBackgroundWorkerShutdown() polyfill (now in core)
* Remove MemoryAccounting_Reset() calls (removed in GP7+)
* Update tuple descriptor attribute access from pointer to direct access

Documentation:
* Rewrite README.md for Apache Cloudberry with updated build instructions

Other improvements:
* Update extension comment to be more descriptive
* Ensure postgres.h is included first in all source files

CI:
* add ic-diskquota to `build-cloudberry.yml` workflow
* For `build-deb-cloudberry.yml`, the installation and configure prefix
  are not consisent, which results in the test error. Will add
  ic-diskquota test back once updating the deb workflow.

See: https://lists.apache.org/thread/1zd80r1hvpwwh5fjd5yqgcc7sr4f27qr
Add diskquota extension license information to comply with Apache
release requirements.

Changes:
- Add diskquota entry to top-level LICENSE file under Greenplum section
- Create licenses/LICENSE-diskquota.txt with PostgreSQL License text
- Add gpcontrib/diskquota/** to pom.xml excludes for apache-rat checks

The diskquota extension is licensed under the PostgreSQL License,
originally developed by Pivotal Software and VMware.
@tuhaihe tuhaihe merged commit 4a607d4 into apache:main Feb 27, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.